Papers with GPT-2 Small
Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck? (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit a puzzling disparity in their formal linguistic competence, even after training on trillions of tokens. |
| Approach: | They pre-train Large Language Models on 100M-token corpora and inject a minimal amount of synthetic data targeting specific linguistic phenomena into the model. |
| Outcome: | The proposed intervention significantly improves model performance in 8 out of the 9 worst-performing BLiMP paradigms. |